Exploiting Competitive Planner Performance

To date, no one planner has demonstrated clearly superior performance. Although researchers have hypothesized that this should be the case, no one has performed a large study to test its limits. In this research, we tested performance of a set of planners to determine which is best on what types of problems. The study included six planners and over 200 problems. We found that performance, as measured by number of problems solved and computation time, varied with no one planner solving all the problems or being consistently fastest. Analysis of the data also showed that most planners either fail or succeed quickly and that performance depends at least in part on some easily observable problem/domain features. Based on these results, we implemented a meta-planner that interleaves execution of six planners on a problem until one of them solves it. The control strategy for ordering the planners and allocating time is derived from the performance study data. We found that our meta-planner is able to solve more problems than any single planner, but at the expense of computation time. 1 Motivation The implicit goal of planning research has been to create the best general purpose planner. Many approaches have been implemented (e.g., partial order, SAT based, HTN) for achieving this goal. However , evidence from the AIPS98 competition 9] and from previous research (e.g., 14]) suggests that none of the competing planners is clearly superior on even benchmark planning problems. In this research, we empirically compared the performance of a set of planners to start to determine empirically which works best when. No such study can be comprehensive. To mitigate bias, we tested unmodiied publically available planners on a variety of benchmark and new problems. As a basis, we included only problems in a representation that could be handled by all of the planners.